Do Subject Matter Experts’ Judgments of Multiple‐Choice Format Suitability Predict Item Quality?

To assemble a high-quality test, psychometricians rely on subject matter experts (SMEs) to write items. However, SMEs are not typically given the opportunity provide input which content standards most suitable for multiple-choice questions (MCQs). In present study, we explored relationship between perceived MCQ suitability standard and associated item characteristics. Prior writing, surveyed each standard. Following field testing, then used SMEs’ average ratings predict characteristics tests. We analyzed multilevel models predicting difficulty (p value), discrimination, nonfunctioning distractor presence. Items were nested within courses standards. There was curvilinear such that very low predictive of easier After controlling difficulty, items with higher had discrimination less likely have one or more distractors. This research has practical implications optimizing test blueprints. Additionally, may use these better prepare coaching during writing.